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This study evaluated the efficacy of the Word Learning Strategies (WLS) supplementary program 
designed to develop upper-elementary students’ vocabulary skills in order to improve reading 
comprehension. The study used a true, group-randomized, experimental design, which randomly 
assigned 46 4th grade classrooms (n=1324 students) from 12 districts to a treatment or control 
group. The results from the first cohort indicate that the program was positively associated with 
gains in students’ vocabulary learning as measured by Word Learning Strategies Test and in 
students’ reading comprehension as measured by Gates-MacGinitie Reading Test, after accounting 
for differences in baseline test scores. The use of the WLS program also led to increases in 
teachers’ awareness of strategies to support their students’ vocabulary and reading 
comprehension. 
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Study Overview 

A significant number of our nation’s students do not develop the level of reading 
proficiency that they need to achieve in school, successfully join the increasingly knowledge- 
oriented workforce, and assist the U.S. in competing in the global economy. Reading is a complex 
process involving multiple interrelated components, and vocabulary is one of the most important of 
these components (Bowers & Kirby, 2009; Carlisle, 2010; McCutchen & Logan, 2011). The Word 
Learning Strategies (WLS) program is a supplementary program designed to develop upper- 
elementary students’ vocabularies in order to improve reading comprehension. The goals of this 
paper are to address: (1) the feasibility of implementing the WLS program in urban elementary 
schools with high numbers of English learners (ELs) and students from low-income backgrounds; 
(2) the potential impact of the WLS program on students’ vocabulary and reading comprehension; 
(3) implications for vocabulary instructional practice; and (4) implications for evaluation theory, 
method, and practice. 

WestEd evaluated and documented the feasibility of implementing the WLS program with a 
focus on: (1) the key components of the WLS program, (2) complementary integration of the WLS 
program into teachers’ existing vocabulary instructional practice, and (3) students’ use of 
generative word learning strategies to understand the meaning of unknown words and concepts 
while reading. 


Description of Word Learning Strategies Program 

WLS is a supplemental program for teaching word learning strategies. The program was 
developed under a Small Business Innovation Research (SBIR) development study awarded by the 
Institute of Education Sciences (IES), and all materials are fully developed and available for 
conducting an efficacy study. The program includes a set of practical, research-based, and 
theoretically sound strategies for inferring the meanings of unknown words that students encounter 
while reading, thereby increasing their ability to derive meaning while reading independently 


(Duffy et al., 1986; Duke & Pearson, 2002; Pressley, Harris, & Marks, 1992; Wharton-McDonald, 
2006; Duke, Pearson, Strachan, & Billman, 2011). Students are taught to use context clues, word 
parts, and the dictionary to learn the meaning of unknown words. Spanish-speaking ELs receive 
additional instruction in using cognates, and all ELs receive instruction in recognizing idioms. 

The basic model for instruction is teacher-led direct explanation with constructivist 
elements, an approach explained below. The program is intended for all students—ELs, average 
learners, above average learners, and less proficient learners. The program prepares teachers to 
teach word learning strategies and to explain to students why the strategies are important for 
reading. Teacher materials include: (1) online tutorials, including videos, to prepare teachers to use 
the materials; (2) a detailed teacher manual with day-by-day lesson plans, a teacher reflection log, 
and instructions for using the online system for supplemental lessons; and (3) presentation 
materials (e.g., slides for overhead projecting, posters, game cards). 

Student materials include activity books, quizzes, and tests. In addition to strategy practice 
with individual words and sentences, larger passages of authentic text are provided so that students 
can practice using these strategies as they would while reading independently. To provide extra and 
differentiated assistance for students who need it, the program includes supplementary, web-based 
instruction and games for each of the strategies, instruction on using cognates (for Spanish- 
speaking ELs), and instruction on recognizing and understanding idioms (for all ELs). 

The program provides 15 weeks of whole-class instruction for a typical 4th or Sth grade 
class, an additional 22 remedial, web-based lessons for students who need more practice, three 
web-based lessons on Spanish cognates for Spanish-speaking EL students, and three web-based 
lessons on idioms for all ELs. The whole-class instruction is delivered three days a week for about 
30 minutes per day throughout the 15-week period. 

The teacher manual includes four main instructional sections. The Word Parts Unit (seven 
weeks) provides lessons for teaching students how to identify and use morphology (inflectional 
suffixes, prefixes, derivational suffixes, roots, and compound words) to derive the meaning of 
unknown words they encounter as they read independently. The Context Unit (five weeks) 
provides lessons for teaching students to infer the meaning of unknown words from linguistic 
context clues (definition, synonym, antonym, and general clues). The Dictionary Unit (one week) 
provides lessons for teaching students to effectively use dictionaries as they are reading to identify 
the meaning of unknown words. The Combined Strategy Unit (two weeks) provides lessons for 
teaching students to combine word parts, context, and dictionary strategies to derive the meaning of 
unknown words. In each lesson plan, the teacher’s guide provides key elements of successful 
instruction: 

1. Key Messages: The points to be emphasized with students during the lesson (e.g., “You 
can use smaller words inside compound words to explain their meanings.”’); 

2. Objectives: A description of what students will be able to do by the end of the lesson 
(e.g., “Define compound word.”’); 

3. Lesson at a Glance: A quick overview of the predictable and consistent lesson structure 
(A. Focus, B. Teach, C. Practice/Apply, D. Wrap Up) with the number of minutes 
needed for each part of the lesson; and 

4. Materials and Equipment: A list of supplies needed for the lesson. 


Each lesson in the guide begins with a brief “Focus Activity” designed to capture students’ 
attention and motivate them to learn. This may be in the form of a quick game, some thought- 
provoking questions, or a brief review. The main instructional activities, which are the bulk of the 


lesson, are the “Teach and Practice/Apply” activities. Time devoted to these activities varies 
depending on where students are in each unit. In the earlier unit lessons, teachers devote more time 
to teaching, modeling, and guiding. As the unit progresses, direct teaching time decreases, and the 
time dedicated to practice and application increases. Assessment occurs every two to three weeks. 
The final part of each lesson is the “Wrap Up” section, during which teachers bring the lesson to a 
close, provide corrective feedback, summarize what students learned, and/or give students a chance 
to reflect on their learning. A speech balloon icon in the teacher’s guide signals the sample teaching 
language that is provided to offer suggestions for explaining strategies, giving directions, posing 
questions, and interacting with students. In addition, a computer monitor icon marks the activities 
that have accompanying video in the web-based teacher training. 

As noted, the pedagogy used in the whole-class instruction is a combination of two widely 
researched and recommended approaches. The first approach—direct explanation of strategies— 
includes: (1) an explicit description of the strategy and when and how it should be used; (2) teacher 
and/or student modeling of the strategy in action; (3) collaborative use of the strategy in action; (4) 
guided practice using the strategy with gradual release of responsibility; and (5) independent use of 
the strategy (Duffy et al., 1986; Duke & Pearson, 2002; Duke, Pearson, Strachan, & Billman, 
2011). The second approach—the use of constructivist elements—is primarily motivated by the 
work of Pressley and his colleagues (Pressley, Harris, & Marks, 1992; Wharton-McDonald, 2006), 
who found that successful use of direct explanation typically involves constructivist elements. 
Those in WLS include: (1) motivating students to use the strategies; (2) discussing with students 
the value of the strategies; (3) providing verbal explanations and collaborative discussion of the 
thinking processes associated with strategy steps; (4) providing extensive feedback and engaging in 
substantial collaborative discussion as students try strategies; and (5) extending instruction and 
practice over a long period of time and across diverse tasks. The inclusion of these constructivist 
elements is further prompted by the importance of motivation as recognized by the National 
Research Council (1999) and reading theorists such as Guthrie, Wigfield, and Perencevich (2004), 
as well as by modern theories of transfer such as those of Engle (2012) and Perkins and Salomon 
(2012). 


Logic Model 

The study’s logic model (Figure 1) posits that implementation of the 15 weeks of WLS 
curriculum, along with its web-based interactive games, will improve students’ use of WLS in 
reading passages, their vocabulary, their reading comprehension, and eventually their school 
achievement. Specifically, the study addressed the following research questions: 


1. Does the WLS intervention increase vocabulary knowledge for 4" graders? 


2. Does the WLS intervention improve reading proficiency for 4" graders? 


Teachers: 


-4th grade teachers attend 4 hours of FD training 
in summer 

-Teachers implement WLS curriculum for 1 weeks 
-Teachers communicate with researchers and WLS 
curriculum experts through emails and conference 
calls as needed 


Students: 
-4th graders are provided with the following WLS 


Assumptions 
* Student difficulties with reading from a lack of knowledge of the words in 
the passage 


* Students’ with low4ncome and EL status face additional bamers to success 


Figure 1. WLS Logic Model 
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Study Method 


In Spring 2016, WestEd conducted a pilot WLS study. The pilot study used a single-subject 
design, in which all participating classrooms took part in the intervention. The intervention 
consisted of five hours of teacher professional development and 12 weeks of whole-class 
instruction of WLS curriculum!. The content of the whole-class instruction of WLS curriculum 


included all curriculum units and associated student activities. 


Participants 


A total of six classroom teachers from two districts, and a total of 180 students from the 
teachers’ classrooms, participated in the study. For the purposes of the pilot study, we did not 
collect student-level demographic information. However, as both of these districts are 
participating in our main study, we do not anticipate that our pilot sample will differ substantially 
from our main sample. The two Bay Area school districts represented a diverse student 
population with 82% and 48% of students receiving free-and-reduced lunch and 58% and 32% of 


students identified as ELs, respectively. 


Measures of implementation fidelity 


Instruments 


In order to monitor and measure fidelity of implementation, we collected weekly teacher 


' Given the timeline, we shortened the intervention to 12 weeks, but still covered all key components of the WLS 


curriculum classroom instruction. 


logs of classroom activity, as well as conducted classroom observations and interviews. 

(1) Teacher Logs: These logs were designed to measure the extent to which participating 
teachers covered WLS concepts and used WLS instructional strategies. The logs were aligned 
with the WLS curriculum to provide a measure of fidelity of implementation. General reporting 
categories on the teacher log included: (a) amount of teaching time devoted to WLS; (b) use of 
various WLS teaching strategies; (c) teachers’ perceptions related to student understanding; and 
(d) questions related to any problems or issues that teachers encountered during implementation, 
including questions on pacing of the lessons and use of supplementary materials for EL students. 

(2) Classroom Observations and Teacher Interviews: Classroom observations were 
conducted in all 6 classrooms. The classroom observations were designed to allow 
documentation of: (a) the WLS components covered in the lesson; (b) resources and equipment 
used; (c) classroom set up; and (d) a snapshot of student activities. Teacher interviews focused 
on: (a) teachers’ use of the WLS curriculum; (b) student engagement and learning; and (c) 
feedback on the WLS training. 


Student measures 

In order to get a broader picture of student achievement, we used separate quantitative 
measures of student knowledge: (1) The Word Learning Strategies Test (WLS Test) and (2) The 
Gates-MacGinitie Reading Test (GMRT). 

(1) The WLS Test is a 34-item test created by the developer of the intervention. It includes 
closed and open-ended items, and assesses student knowledge of prefixes, suffixes, context cues, 
as well as the Word Parts Strategy, the Dictionary Strategy, and the Combined Strategy. The 
measure also assesses students’ ability to apply the Word Parts, Context, and Dictionary 
Strategies to highlighted words presented in the context of short stories. Thirty-five percent of 
the assessment tests knowledge, and 65% tests application. Data collected during the SBIR pilot 
test of the WLS intervention indicate that the instrument has good reliability. Specifically, 
Cronbach’s alpha for the entire instrument ranged from 0.875 at pre-test to 0.921 at post-test. 

(2) The VASE Assessment (Scott, Flinspach, Vevea, & Castaneda, 2012) is a 24-word 
assessment. The development of the VASE Assessment was funded through an IES Goal 5 
grant. The assessment: 1) tests students’ familiarity with grade-level vocabulary in math, 
science, social studies, and language arts; 2) provides diagnostic information about specific 
aspects of their word knowledge; and 3) measures their vocabulary growth over the school year 
and across 4th and 5th grades. VASE results identify strengths and weaknesses in the breadth 
and depth of students’ academic vocabularies. Results from correlation analyses, exploratory 
factor analysis, and confirmatory analysis indicated that the VASE Assessment has good 
convergent and construct validity. The internal reliability coefficient was 0.95. 

(3) The GMRT (MacGinitie, MacGinitie, Maria, & Dreyer, 2002) is a series of 
standardized, multiple choice, norm-referenced tests of reading achievement that can be 
delivered in a paper/pencil format or online. The GMRT for grades 3-12 includes two subtests— 
vocabulary and comprehension. The difficulty level of the questions on the GMRT progresses 
from easy in the beginning to difficult at the end. Each level of the GMRT is designed to 
accurately measure performance across a range of reading levels. Kuder-Richardson Formula 20 
(KR-20) was utilized to assess the reliability index for the subtests. Internal reliability 
coefficients were 0.80 for the vocabulary subtest and 0.90 for the comprehension subtest. 


Results 


Program Implementation 

Prior to implementing the WLS supplemental curriculum, the six pilot teachers participated 
in approximately 5 hours of professional development provided by the developer of WLS and by 
WestEd. The developer of WLS conducted a two-hour face-to-face training with teachers that 
provided background information about vocabulary development, instruction, and the use of 
WLS. Teachers then participated in an hour-long online module on their own time. Finally, 
teachers reconvened with the WLS developer and WestEd researchers for the final two hours of 
professional development. In this face-to-face session, the WLS developer discussed the online 
training and topics related to curriculum implementation (e.g., following the teacher manual, 
dosage, pacing). The WestEd research team also provided an hour-long module describing the 
research tasks associated with the study and how teachers were to collect student assessment 
data. 

Implementation fidelity 

In order to monitor teachers’ implementation of WLS—particularly as it related to dosage 
and pacing—pilot teachers completed a weekly teacher log. Each teacher completed 100% of his 
or her logs. Based on these logs, we were able to determine whether each teacher completed all 
of the WLS lessons. The curriculum recommends 15 weeks to get through all of the lessons, but 
we asked the pilot teachers to complete all of the lessons in 12 weeks due to time contraints that 
semester. Teachers completed the 12 weeks of instruction by the very end of the semester. 
Under ideal conditions, the curriculum would be implemented in the fall, instead of the spring. 
In this way, teachers would be able to easily implement all of the lessons and help students to 
generalize these strategies across content areas for the remainder of the school year. This is how 
WLS is being implemented for the main study. Nevertheless, the pilot teachers on average were 
able to complete 90% of the lessons, ranging from 73% to 100% by teacher. The teacher logs 
also suggested that teachers were implementing WLS for the appropriate duration and at the 
recommended pace. Collectively, the teachers implemented WLS on average three days per 
week for 29 minutes each day, which is the recommended dosage. There did seem to be some 
discrepancy as it related to pacing, at least for one teacher. This teacher skipped 12 lessons and 
repeated 13 lessons. Although it would be expected that teachers would repeat some lessons for 
review purposes—the range for the remaining teachers was 0-6 repeated lessons—it was unusual 
that so many lessons were repeated while a large number were also skipped. In order to address 
issues related to pacing for the main study, researchers created pacing guides based on each 
district calendar to help teachers monitor and plan for completion of all the lessons. 


Finally, the teacher logs allowed us to examine students’ progress monitoring. Although all 
six teachers reported that on average 96% of their students mastered the core concepts taught for 
each lesson, they only administered an average of 31% and 28% of the quizzes and tests 
respectively. Further, teachers on average only reviewed the student activity books 47% of the 
time. As such, it is unclear how teachers were able to estimate that so many of their students 
mastered the core concepts. For the main study, we emphasized the use of quizzes and tests and 
let teachers know that researchers would be collecting their quizzes and tests for review. 


Integration of WLS into existing instructional practices 

In addition to the teacher logs, researchers conducted direct classroom observations of all six 
participating pilot teachers’ implementation of WLS. The WestEd research team developed an 
observation protocol in collaboration with the WLS developer to make sure all relevant aspects of fidelity 
specific to WLS were captured. Once the protocol was developed, teams of two conducted the 


observations. Each team consisted of a co-investigator and a research assistant. In this way, we were able 
to begin to establish inter-rater reliability for the main study and to determine what revisions to the 
observation protocol, if any, would be necessary. In addition to collecting a running narrative of the 
lesson observed and its length in minutes, the research team also evaluated two overall dimensions of 
fidelity: 1) the degree to which the teacher adhered to the instructional components specific to WLS, and 
2) the overall quality of instruction. There were five specific items that were scored in a binary manner 
(i.e., yes it was observed or no it wasn’t) for each over-arching dimension. Related to the specific 
instructional components of WLS, researchers observed whether: a) all lesson components were taught, 
b) appropriate curriculum materials were used, c) the teacher followed the WLS manual, d) the teachers’ 
subject matter knowledge reflected adequate understanding of the content, and e) the lesson closing was 
implemented as intended. Related to more general teaching quality, researchers observed whether: a) 
there were any barriers to implementation, b) pacing was appropriate, c) students were engaged, d) 
engagement was distributed (1.e., teacher called on a variety of students), and e) the overall quality of the 
lesson was positive. 

In scoring these 10 specific items, the six teachers’ combined average fidelity was 85%, ranging 
from 60% to 100%. Overall, this level of fidelity suggests that teachers were able to learn the WLS 
curriculum and implement it correctly without a laborious amount of professional development. 
Specifically, all teachers were successfully able to teach all of the WLS lesson components, use the 
appropriate materials, display adequate subject-matter knowledge, and keep students engaged. As such, 
all six teachers’ overall presentation was considered positive by the researchers. The area most 
problematic was pacing, where three out of the six teachers either rushed the lesson or took too long. 
Although the average number of minutes was 31 and the intended number of minutes was 30, the 
variation among teachers ranged from 20 to 40 minutes. 

After each observation, a brief teacher interview was conducted to obtain feedback about the WLS 
curriculum. In general, the teachers felt the curriculum was easy to learn and to implement, and was 
beneficial to students, particularly to their ELs. Teachers indicated that their students were able to easily 
master the skills, and suggested that we move the project to fourth grade. This was a unanimous 
recommendation by the six pilot teachers. They felt it would be better to introduce the word learning 
strategies at fourth grade to maximize students’ time for integrating those skills across content areas for 
the last two years of elementary school. Even though the fifth grade students during the spring semester 
seemed to benefit from the curriculum, as indicated in the results below, we agreed with the teachers 
about maximizing the benefit of the WLS by moving it to fourth grade. As such, the main study is being 
conducted at fourth grade instead of fifth grade as originally planned. 


Initial Findings: Students’ Vocabulary and Reading Comprehension 

The results from an analysis of quantitative assessment data indicate that the intervention was 
positively associated with gains in vocabulary as measured by the WLS, VASE, and GMRT assessments. 
Student gains on the WLS test (mean percent gain=14.29) were significant at the 0.001 level. Gains on 
the VASE assessment (mean converted score gain=1,032.84) were also significant at the 0.001 level. 
Gains on the Vocabulary subtest of the GMRT (mean gain=1.43) were significant at the 0.01 level. Gains 
were not statistically significant for the GMRT Comprehension subtest, or the GMRT composite score 
(see Table 1). 


Table 1. Mean Student Pre- and Post-test Scores 


Pretest Mean Score Post-test Mean Maximum Possible 


Score Score 

WLS Percent Correct 68.91 83.20*** 100 
(19.21) (15.46) 

VASE Converted Scores 5892.58 6925.42*** 9000 
(592.01) (648.92) 

GMRT Vocabulary 25.36 26.79** 45 
(9.54) (8.20) 

GMRT Comprehension 27.93 28.24 48 
(10.55) (9.93) 

GMRT Total 53.29 54.99 93 
(19.32) (17.35) 


**Gains significant at aw = 0.01 
***Gains significant at aa = 0.001 


Pre- and post-test correlations between the assessments indicate a moderate to strong 
correlation between student performance on the GMRT, VASE, and WLS, suggesting that student 
performance across measures was relatively consistent (Tables 2 & 3). These strong correlations 
are encouraging as we plan to use these assessments in the future WLS study to measure impact. 
Results for each measure are described in detail below. 


Table 2. Pre-test Correlations 


GMRT GMRT GMRT V7.\) 3 
Total Comprehension Vocabulary Converted 
Score Score Score Score 


100 
0.65 0.64 0.61 1.00 


WLS Percent Correct (N=128) 0.77 0.75 0.72 0.48 


Table 3. Post-test Correlations 


GMRT GMRT VASE 
Comprehension Vocabulary Converted 
Score Score Score 


100 
ose ost 3.00 
0.63 0.58 0.63 1.00 


WLS Percent Correct (N=125) 0.73 0.72 0.67 0.52 


WLS test results 
The greatest relative gains were seen for the proximal WLS test, with the average number of 
items answered correctly rising by a statistically significant 14.29% (t=11.29; p=0.000) (Figure 2). 


These results indicate that students understood the steps and reasoning behind the various word- 
learning strategies taught in the WLS curriculum, and were able to apply these strategies to the 
WLS vocabulary words. While the proximal nature of this test means that these results cannot be 
extended to general vocabulary learning, it serves as an important indication that students are 
learning the skills intended by the curriculum. 
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Figure 2. WLS: Pre- and Post-test Mean Percent Correct 


VASE assessment results 

Students performed significantly better on the VASE post-test than on the pre-test, with a mean 
converted score gain of 1,032.84. Converted scores are derived from a scale provided by the 
assessment developer where the minimum score is 1,000, the maximum score is 9,000, and the 
normed mean is set at 5,000. The scale is designed to allow for comparisons across test forms, and 
across grade levels. As shown in Figure 3, students in the pilot study already performed above 
average for their grade level on the VASE pre-test, but still scored significantly higher on the post- 
test (t=19.62; p=0.000). In this study, the VASE assessment serves as a general measure of 
vocabulary knowledge, using words that do not necessarily align with the words taught in WLS. 
The results from this assessment indicate increased student vocabulary knowledge from the time of 
the baseline assessment, which may be associated with learning gains from the WLS curriculum. 
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Figure 3. VASE: Pre- and Post-test Mean Converted Scores 


GMRT assessment results 

On the GMRT assessment overall, students did not show significant gains from pre- to post- 
test. An analysis of the two subtests individually, however, indicates differences in gains between 
the vocabulary and comprehension subtests (Figure 4). Student scores did not change significantly 
on the GMRT comprehension subtest. For the GMRT vocabulary subtest, however, student gains 
were significant (t=3.01; p=0.003). In summary, the pilot study results indicate that the WLS 
intervention is not only associated with gains on proximal vocabulary test (WLS test), but also 
associated with gains on distal vocabulary tests (VASE and GMRT vocabulary subtest). 
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Figure 3. GMRT: Pre- and Post-test Mean Converted Scores 


Exploring comprehension outcomes 

It is interesting to note that students in our pilot showed significant gains on the GMRT 
vocabulary subtest, but not on the comprehension subtest. Other researchers have explored the 
relationship between vocabulary interventions and comprehension outcomes, taking into account 


10 


various factors, such as instruction type and outcome measure (e.g., general-word test vs. taught- 
word test). In particular, Wright and Cervetti (2016) conducted a systematic review of such 
research ranging from 1965 to 2014, providing a rigorous synthesis of findings. They examined 
research that focused on strategy-based instruction and employed a generalized comprehension 
outcome measure that tested students on words that were not necessarily taught during instruction. 
Although the authors concluded that there was little research to date that showed vocabulary 
instruction impacting student comprehension, they found that 2 of 7 studies showed a statistically 
significant impact on comprehension gains. The two studies that focused specifically on low- 
achieving students found that low-achieving students who received interventional vocabulary 
instruction showed significant gains compared to a control group (Lubliner & Smetana, 2005), or 
caught up to more advantaged peers (Nelson & Stage, 2007). 

In light of the research, we conducted an exploratory analysis of student GMRT scores by 
baseline performance, and found that GMRT comprehension post-test scores were significantly 
improved for students scoring in the lowest 50% of that respective pre-test (Table 4), suggesting 
that lower-achieving students may reap additional comprehension benefits from vocabulary 
instruction as compared to their higher-achieving peers. These results are encouraging for the 
continuing implementation of the WLS curriculum in diverse schools with substantial populations 
of low-income students and ELs. As we move forward with the future study, it will be imperative 
to analyze impact on comprehension outcomes, taking into account student achievement level at 
baseline. 


Table 4. GMRT Score Gains by Baseline Quartile 


Mean Gain (SD) 
Baseline Quartile 1* Quartile vue elelaal(= 3" Quartile 4** Quartile 


5.00 (5.04)*** 3.66 (4.93)*** 2.24 (5.22)*** 1.4 (5.14)** 
lal AM easel N=34 N=35 N=29 N=31 
4.61 (7.32)*** 2.98 (7.55)** 1.49 (7.53) 0.34 (7.37) 
GMRT Comprehension N=36 N=29 N=35 N=29 
8.70 (11.02)***  6.41(10.57)*** 3.59 (10.66)** 1.11 (10.43) 
ul) N=37 N=28 N=34 N=30 


**Gains significant at awa = 0.01 
***Gains significant at aa = 0.001 


Implications and Future Study 

Importance of Communication with Teachers 

Throughout the pilot study, the WestEd research team maintained frequent communication with 
the six teachers implementing the WLS curriculum. To facilitate communication with the teachers, 
the research team created a shared email account which was consistently monitored to ensure 
prompt replies to teacher inquiries. Timely responses to questions or concerns were critical to 
prevent teacher confusion about implementation of the curriculum, which could have affected 
implementation fidelity or resulted in the collection of unreliable data. 

The weekly teacher logs allowed researchers to examine whether teachers were implementing 
the curriculum as intended, enabling researchers to offer support to teachers who were 
experiencing issues with implementation. In addition to using teacher logs to collect information 
about the number of lessons completed or the time spent per lesson, the logs also provided insight 
on the level of teacher engagement. WestEd researchers aimed to maintain a high level of teacher 
engagement to minimize participant fatigue and prevent study attrition. Moving into the main 


study, weekly newsletters will be used to inform teachers of study updates, including additional 
resources, helpful tips, and stipend delivery. Researchers will continue to emphasize the 
importance of the teachers’ role throughout the study and acknowledge their efforts in contributing 
to a body of research around elementary vocabulary instruction. 


Looking Forward: Future RCT Study 

The results of the pilot study indicate that by focusing on one facet of vocabulary instruction— 
teaching word learning strategies—the WLS curriculum responds to the need for teachers to be 
equipped with powerful yet straightforward ways to help their students improve their vocabularies, 
which in turn has the potential to improve their reading comprehension, their motivation, and their 
overall success in school. The WLS curriculum provides clear guidance and appropriate materials 
so that teachers can successfully implement the curriculum and track the progress of their students 
as they do so. To further address the efficacy of the WLS curriculum, we are currently conducting a 
rigorous randomized controlled trial (RCT) involving 90 teachers. 

Teachers are randomly assigned to the control group or the WLS treatment group. Teachers 
randomly selected to be in the control condition implement their usual teaching practices. The 
control teachers are informed that they are guaranteed the opportunity to receive the WLS 
curriculum and professional development in the summer of 2017. Thus, the control group serves as 
a business-as-usual condition, representing the type of instruction students would normally receive 
at schools. Teachers randomly selected to be in the treatment condition implement the WLS 
curriculum. To measure the fidelity of treatment implementation, researchers use multiple sources 
(teacher training sign in, teacher logs of implementing the curriculum, student digital lesson 
analytics, classroom visits) to track each key component of the WLS treatment as listed in the logic 
model (Figure 1). The key components are: (1) teacher professional development training on WLS 
curriculum, (2) support to teachers on how to implement the curriculum, (3) providing students 
with 15 weeks of classroom instruction on Word Parts, Context, Dictionaries, and Combined 
Strategies, (4) providing students with web-based interactive games to reinforce corresponding 
classroom instruction, and (5) providing ELs with additional web-based interactive games to meet 
their needs. In addition to the GMRT, WLS, and VASE assessments, the RCT study will also 
collect pre- and post-distal measures: student scores on the Smarter Balanced assessment for 
English Language Arts (ELA) and California English Language Development Test (CELDT). The 
primary hypothesis-testing analyses will involve fitting conditional mixed effects ANCOVA 
models (hierarchical linear modeling—HLM or multilevel models), with an additional term to 
account for the nesting of students within teachers. Potential fixed effects include treatment group 
(intervention), baseline (pretest) measures of literacy proficiency, and other observed covariates 
such as free/reduced-price lunch status, EL status, and gender. 

As we move forward with the main RCT study, we draw on lessons learned from the pilot 
study in order to ensure a rigorous research design and study implementation. We look forward to 
analyzing the results from the main study, and understanding the impact that the WLS curriculum 
has on student vocabulary and reading comprehension skills in urban elementary schools. 


Acknowledgement 
This project is supported by a grant from the U.S. Department of Education, Institute of 
Education Sciences (IESR305A150108). Any opinions, findings, conclusions or 
recommendations are those of the authors and do not necessarily reflect the views of the Federal 
Government. 


12 


References 

Bowers, P. & Kirby, J. (2009). Effects of morphological instruction on vocabulary 
acquisition. Reading and Writing: An Interdisciplinary Journal, 23(5), 515-537. 

Carlisle, J. F. (2010). An integrative review of the effects of instruction in morphological 
awareness on literacy achievement. Reading Research Quarterly, 45, 464-487. 

Duke, N. K. & Pearson, P. D. (2002). Effective practices for developing reading 
comprehension. In A. E. Farstrup & S. J. Samuels (Eds.), What research has to say 
about reading instruction (3rd ed., pp. 204-242). Newark, DE: International Reading 
Association. 

Duke, N. D., Pearson, P. D., Strachan, S. L., & Billman, A. K. (2011). Essential elements 
of fostering and teaching reading comprehension. In S.J. Samuels, & A. E. Farstrup 
(Eds.), What research has to say about reading instruction, (4th ed., pp. 51-93). 
Newark, DE: International Reading Association. 

Duffy, G.G., Roehler, L.R., Meloth, M.S., Vavrus, L.G., Book, C., Putnam, J., et al. 
(1986). The relationship between explicit verbal explanations during reading skill 
instruction and student awareness and achievement: A study of reading teacher 
effects. Reading Research Quarterly, 21(3), 237-252. 

Engle, R.A. (2012). The resurgence of research into transfer: An introduction to the final 
articles in the transfer strand. Journal of the Learning Sciences, 21, 347-352. 

Guthrie, J. T., Wigfield, A., & Perencevich, K. C. (Ed.). (2004). Motivating reading 
comprehension: Concept-Oriented Reading Instruction. Mahwah, NJ: Erlbaum. 

MacGinitie, W., MacGinitie, R., Maria, K., & Dreyer, L. (2002). Gates-MacGinitie 
Reading Tests, 4th Edition Technical Report. Itasca, IL: Riverside. 

McCutchen, D. & Logan, B. (2011). Inside incidental word learning: Children's strategic 
use of morphological information to infer word meanings. Reading Research 
Quarterly, 46, 334-349. 

Perkins, D.N. & Salomon, G. (2012). Knowledge to go: A motivational and dispositional 
view of transfer. Educational Psychologist, 47(3), 248-258. 

Pressley, M., Harris, K. R., & Marks, M. B. (1992). But good strategy instructors are 
constructivists! Educational Psychology Review, 4, 3-31. 

Scott, J.A., Flinspach, S.L., Vevea, J.L., & Castaneda, R. (2012). Presentation at IES. 

Wharton-McDonald, R. (2006). The need for increased comprehension instruction. In M. 
Pressley (Ed.), Reading instruction that works: The case for balanced teaching (pp. 
293-346). New York: Guilford. 

Wright, T.S. & Cervetti, G.N. (2016) A systematic review of the research on vocabulary 
instruction that impacts text comprehension. Reading Research Quarterly, 0(0), 1-24. 


13 


